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I.  INTRODUCTION 


Simulation  models  that  involve  many  factors  (i.e.,  input  variables) 
necessitate  large  experimental  programs.  Often,  though,  it  is  expected 
that  only  a  relatively  small  number  of  the  factors  have  an  appreciable 
effect  on  the  response  (i.e.,  output  variable).  In  order  to  reduce  costs 
by  concentrating  the  major  experimental  effort  on  the  most  Important 
factors,  a  frequent  desire  is  to  screen  the  factors  in  the  hope  of  selecting 
the  important  ones  for  further,  more  detailed  study  in  subsequent 
experimentation.  In  such  a  situation,  the  simulation  user  would  like  to 
conduct  a  preliminary  experiment  aimed  at  determining  the  size  and 
composition  of  the  subset  of  important  factors. 

Although  this  report  places  factor  screening  in  a  simulation  context, 
screening  experiments  can  arise  in  virtually  any  field  of  scientific  research. 
In  fact,  much  of  the  existing  screening  methodology  is  a  result  of  research 
in  the  arena  of  industrial  experimentation.  Overall,  however,  the  problem 
has  yet  to  be  resolved  satisfactorily  or  studied  systematically.  Thus,  no 
definitive  guidelines  for  selecting  a  screening  strategy  have  been  developed, 
although  a  number  of  general  suggestions  have  been  made.  [See  Kleijnen 
(1975)  for  a  review.] 

To  a  large  extent,  choosing  a  screening  procedure  is  governed  by  the 
number  of  simulation  runs  available  for  screening.  When  affordable, 
screening  strategies  based  on  standard  orthogonal  experimental  designs 
are  recommended.  However,  these  designs  require  more  runs  than  factors 
to  be  screened,  a  luxury  seldom  provided  in  the  simulation  environment 


because  of  budget  and  running  time  limitations.  Thus,  screening  methods 
based  on  nonstandard  designs  which  allow  for  fewer  runs  than  factors  to 
be  screened  are  usually  of  greater  practical  interest. 

In  a  previous  Desmatics  technical  report.  Smith  and  Mauro  (1980) 
reviewed  the  major  classes  of  screening  designs,  discussed  possible 
performance  criteria  for  systematically  evaluating  screening  methods, 
and  proposed  two  screening  strategies  for  further  research,  namely, 
two-stage  group  screening  and  a  random  balance/Plackett-Burman  (RB/PB) 
two-stage  combination  strategy.  Both  proposed  strategies  are  intended 
for  situations  in  which  the  available  number  of  computer  runs  is  less 
than  the  number  of  factors  to  be  screened.  More  recently,  Mauro  and 
Smith  (1980)  formally  examined  and  reported  the  performance  of  the 
two-stage  group  screening  method.  This  report  presents  a  parallel 
analysis  for  the  RB/PB  screening  method. 


II .  PRELIMINARY  DISCUSSION 


In  screening  experiments  a  relatively  small  number  of  factor  levels 
is  generally  employed.  Usually  two  levels,  designated  high  (+1)  and 
low  (-1),  of  each  factor  are  sufficient.  Consequently,  most  screening 
designs  are  two-level  designs.  However,  in  a  two-level  experiment  it  is 
appealing  to  consider  only  designs  which  have  an  equal  number  of  runs 
at  the  high  and  at  the  low  levels  of  a  factor.  That  is,  for  a  design 
employing  N  (even)  runs,  each  column  of  the  design  matrix  would  consist 
of  N/2  +l's  and  N/2  -l’s.  Estimates  of  factor  effects  are  then  uncon¬ 
founded  with  the  overall  mean  effect,  which  can  be  represented  by  a 
column  of  N  +l's. 

In  a  random  design,  a  random  sampling  process  is  used  to  choose 
all  or  some  of  the  elements  of  the  design  matrix.  In  generating  the 
design  matrix,  various  sampling  schemes  can  be  employed.  However,  as 
indicated,  for  a  two-level  situation  it  is  desirable  to  balance  the 
design.  Consider,  therefore,  a  design  matrix  in  which  the  assignment 
of  +l's  and  -l’s  in  each  column  is  made  randomly  so  that  all  possible 
configurations  of  N/2  +l's  and  N/2  -l’s  (there  are  C^^  ln  all)  are 
equally  likely,  with  each  column  receiving  an  independent  randomization. 

An  experimental  design  that  is  a  realization  of  such  a  sampling  plan  is 
called  a  random  balance  design.  Note  that  N  can  be  determined  independently 
of  the  number  of  factors  to  be  screened.  This  is  not  true  of  more 
orthodox  factorial  designs. 


Random  balance  designs  have  generated  much  heated  discussion  as 
to  their  merits  and  demerits  [e.g.,  Bunde  (1959),  Satterthwaite  (1959), 
and  Youden,  Kempthorne,  Tukey,  Box,  and  Hunter  (1959)].  The  main 
objection  is  that  they  confound  factors  to  a  random  degree.  Furthermore, 
there  are  no  specific  techniques  for  analyzing  random  balance  data. 
Typically,  each  factor  is  considered  separately,  and  some  standard 
statistical  analysis  is  applied  to  test  for  the  presence  of  factor 
effects.  Anscombe  (1959)  suggested  using  the  ANOVA  F-test,  Welch's 
Randomization  Test,  or  Tukey' s  Randomization  Test.  These  procedures 
all  admit  to  a  relatively  quick  and  simple  statistical  analysis. 
Conceivably,  more  specialized  techniques  could  be  devised  and  employed, 
but  at  the  expense  of  analysis  simplicity.  In  this  report,  a  standard 
F-test  applied  separately  to  each  factor  will  be  adopted  as  the  method 
of  analysis  for  random  balance  data. 

To  provide  the  same  statistical  basis  to  compare  and  assess 
screening  strategies,  it  is  necessary  to  assume  a  common  model  to 
underlie  the  simulation  responses.  In  screening,  it  generally  suffices 
to  assume  the  first-order  model  given  by 


'  e°  V/1*11  +  €‘ 


(2.1) 


where 


th 

(1)  y^  is  the  value  of  the  response  in  the  i  simulation  run, 

(2)  K  is  the  total  number  of  factors  to  be  screened,  each  of  which 
is  at  two  levels, 

th 

(3)  x^j  is  +1  or  -1  provided  the  j  factor  is  set  at  its  high  or 


th 


low  level,  respectively,  for  the  i —  simulation  run. 
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th 

(4)  3^  is  the  (linear)  effect  of  the  j  factor, 
and  (5)  the  error  terms,  £^,  are  independent  with  common  distribution 
N(0,a2) . 

For  a  given  set  of  values  for  3p  32»****  3^,  and  a  in  model  (2.1), 
the  performance  of  a  screening  method  can  then  be  evaluated.  However,  to 
keep  this  study  manageable  the  following  additional  conditions^  will  be 
imposed: 


(2')  there  are  a  total  of  K  factors  to  be  screened,  k  1 

(unknown)  of  which  are  active  (i.e.,  have  a  nonzero  effect) 
and  (K  -  k)  of  which  are  inactive  (i.e.,  have  a  zero  effect), 
(4')  all  active  factors  have  the  same  absolute  linear  effect,  A  >  0, 


th 

A,  if  the  j  factor  is  active 
th 

0,  if  the  j  factor  is  inactive, 


and  (5')  the  simulation  response  is  observed  without  random  error  (i.e.,  0*0). 

By  (4’)  all  active  factors  are  regarded  symmetrically.  Furthermore, 

(5')  is  equivalent  to  assuming  that  A  is  so  large  compared  with  a  that  a 
can  be  effectively  ignored.  A  forthcoming  report  will  examine  the  performance 
of  the  RB/PB  strategy  when  this  assumption  is  relaxed. 
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III.  THE  RB/PB  STRATEGY 


The  first  stage  of  the  RB/PB  strategy  is  a  K  factor  random  balance 
(RB)  design  employing  N  (even)  runs,  while  the  second  is  a  Plackett-Burman* 

(PB)  design  follow-up  experiment.  A  factor  reaches  stage  two  only  if  it 
has  a  significant  effect  in  stage  one.  Because  PB  designs  are  orthogonal, 
the  second  stage  will  filter  out  any  confounding  between  the  factors 
carried  over  from  the  RB  design.  To  be  classified  as  active,  a  factor 
must  test  significant  in  the  PB  design,  which  can  be  analyzed  by  the 
usual  analysis  of  variance  procedures  for  factorial  experiments.  Factors 
not  reaching  stage  two,  however,  should  be  held  at  a  constant  level  so 
that  the  second-stage  estimate  of  a  factor  effect  is  not  biased  by  any 
effects  of  active  factors  not  reaching  this  stage,  [cf.  Montgomery  (1979).] 

The  number  of  second  stage  runs,  M,  will  depend  on  the  number  of  factors, 

S,  carried  over  from  stage  one.  In  general,  the  smallest  PB  design  having 

at  least  S+l  runs  is  chosen.  Thus,  if  B(x)  denotes  the  smallest  integer 

larger  than  x  that  is  a  multiple  of  four,  then  M  =  B(S).  Accordingly, 

the  total  number  of  runs,  R,  that  the  two  stages  require  will  be  N  +  M  *  N  +  B(S). 

Note  that  although  N  must  be  specified  prior  to  experimentation,  M,  and 

consequently  R,  is  random. 

As  Watson  (1961)  pointed  out,  the  objectives  of  a  factor  screening 
strategy  are: 


^Plackett-Burman  designs  are  two-level,  orthogonal,  fractional  factorial 
designs  for  studying  up  to  (4m-l)  factors  in  4m  runs.  See  Plackett  and  Burman 
(1946). 
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(a)  to  detect  as  many  of  the  active  factors  as  possible, 

(b)  to  declare  active  as  few  inactive  factors  as  possible, 
and  (c)  to  achieve  these  aims  with  as  few  runs  as  possible. 

Therefore,  the  extent  to  which  the  RB/PB  strategy,  or  any  other  screening 
strategy,  satisfies  the  above  objectives  is  a  measure  of  its  performance 
as  a  screening  strategy.  However,  since  objectives  (a)  and  (b),  which 
deal  with  factor  classification,  and  objective  (c),  which  deals  with 
testing  cost,  conflict,  some  balancing  of  objectives  will  be  required 
to  compare  competing  strategies. 

Because  of  the  additive  structure  of  model  (2.1)  and  the  resolution 
of  PB  designs,  a  factor  that  reaches  stage  two  will,  under  the  assumption 
of  no  random  error  (o=0) ,  be  classified  correctly.  Thus,  under  the 
assumptions  made  here,  objective  (b)  is  trivially  satisfied  in  the  RB/PB 
strategy  since  no  inactive  factor  will  be  raisclassif ied.  Performance 
evaluation  can  focus,  therefore,  on  objectives  (a)  and  (c).  Moreover, 
the  number  of  active  factors  classified  correctly.  A,  will  correspond  to 
the  number  of  active  factors  that  test  significant  in  stage  one. 

As  a  measure  of  efficiency  for  detecting  the  active  factors,  define 

E(F)  =  E(A)/k  .  (3.1) 

E(F)  is  normalized  so  that  0  £  E(F)  1;  efficiency  closer  to  one 
indicates  better  performance  on  the  average.  Regarding  the  total  runs 
that  a  strategy  requires. 


E(Q)  *  E(R)/B(K) 


(3.2) 


can  be  used  as  a  measure  of  the  expected  relative  testing  cost  of  a 
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screening  strategy  (relative  to  the  runs  required  by  a  PB  design  for  K 
factors.)  A  smaller  value  of  E(Q)  indicates  better  performance  on  the 
average.  It  is  imperative,  however,  that  both  (3.1)  and  (3.2)  be 
considered  jointly  in  assessing  the  overall  performance  of  a  RB/PB 
strategy.  In  many  ways  the  problem  is  akin  to  the  testing  of  a  statistical 
hypothesis  in  which  the  sample  size  is  desired  small,  but  the  corresponding 
power  is  desired  large. 

Only  if  one  RB/PB  strategy  has  both  a  smaller  E(Q)  and  a  larger  E(F) 
than  another  can  the  first  be  said  to  be  definitely  better  than  the  second. 
For  example,  in  Figure  1  it  is  clear  that  strategy  A  is  superior  to  B  and 
C.  However,  in  comparing  A  with  D,  superiority  depends  on  possible  trade¬ 
offs  in  E(F)  and  E(Q)  the  analyst  is  willing  to  make  in  a  given  screening 
application.  A  cost  function  could,  of  course,  be  defined  in  terms  of 
E(F)  and  E(Q)  to  compare  these  two  strategies. 

For  simplicity,  suppose  that  the  K  separate  F-tests  performed  in 
stage  one  are  each  conducted  at  the  same  level  of  significance,  a.  Of 
interest,  therefore,  is  how  a  and  N  affect  performance,  as  defined  by 
(3.1)  and  (3.2).  To  establish  a  notation,  let  RP(c,a)  denote  the  RB/PB 
strategy  in  which  N  =  cK,  0  <  c  <  1,  and  let  I  denote  the  number  of 
inactive  factors  that  test  significant  in  stage  one.  Since  S  =  A  +  I  and 
B(x)  *  x  +  2.5  ,  it  follows  that\ 

E (R)  s  cK  +  E (A)  +  E(I)  +  2.5  .  (3.3) 

Thus,  to  evaluate  (3.1)  and  (3.2)  It  suffices  to  evaluate  E(A)  and  E(I), 
both  of  which  depend  on  the  testing  characteristics  of  the  K  separate 

^-Jote  that  |b(x)  -  (x  +  2.5)|  ^  1.5;  hence,  the  approximation  in 

(3.3)  can  differ  from  E(R)  by  at  most  1.5  . 


F-tests  in  the  RB  design. 

In  a  random  balance  design  employing  N  (even)  runs,  let 
5C  *  [Xq,  jc^,  . . . ,  x^]  denote  the  N  x  (K+1)  design  matrix  where 
Xq  is  a  (Nxl)  vector  consisting  entirely  of  +l's  and  x^ (j>l)  is  a  (Nxl) 
vector  consisting  of  a  random  arrangement  of  N/2  +l's  and  N/2  -l's. 
Assuming  that  0*0  in  model  (2.1),  £  =  X£  where  j3  is  a  (K+1)  x  1  vector 
of  unknown  parameters  and  ^  is  a  (Nxl)  vector  of  responses. 

The  simple  least  squares  estimator  of  3^  (j_^l)  is  given  by 


ej  ’  t5+j  -  9-i)/2 


(3.4) 


where  y+j  (y_j)  average  value  of  the  response  over  the  N/2  runs 

th 

at  the  high  (low)  level  of  the  j  factor.  In  matrix  terms 


Thus, 


and 


0J  -  (x,WN  =  . 


~  K 

E(3.)  =  (1/N)  L  6,E(x  'x  ) 
J  i=l  1  I  -1 


£ 

V(3.)  -  (1/N2)  Z  3  2  V(x. 'x  ) 
j  i=1  i  “j  ~i 


(3.5) 


(3.6) 


(3.7) 


where  equation  (3.7)  makes  use  of  the  fact  that"  Cov(x  'x,,  x.  'x  )  =  0 

— j  — i  — j  — m 

when  i  m.  To  evaluate  these  moments  a  brief  digression  is  taken. 

If  Z  is  a  discrete  random  variable  such  that 


P(Z=z)  =< 


r  2/  2r 

z  I  /  I  r  |  z  =  0,  1,  2,  . . . ,  r 

otherwise, 

write  Z  ~  H(r).  The  H(r)  family  is  a  unimodal,  symmetric  subfamily 
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of  the  hypergeometric  family  of  distributions.  Furthermore,  if 
Z  ~  H(r),  then  E(Z)  ■  r/2  and  V(Z)  •  r2/4(2r  -  1).  Now  a  key  result 


is  that,  for  i  +  j,  (x^'x^  +  N)/4  ~  H(N/2).  Hence, 

E(V '  bj  ■ 

(3.8) 

2 

V(3j)  =  (W  -  3j  )/(N  -  1)  , 

(3.9) 

A  A 

and  Cov(3i,  3j)  =  3±3j / <N  -  1)  ,  i  ^  j. 

(3.10) 

K  2 

where  W  =  E  3  .  Of  course,  only  active  factors  make  a  nonzero 

j=l  J 

contribution  to  W.  When  i  =  j ,  =  2L£ *— i  =  N  • 

A 

The  simple  least  squares  estimator  of  3 ^ ,  3  ^ ,  is  therefore  an 
unbiased  estimator.  However,  an  obvious  drawback  of  analyzing  factors 
separately  is  that  the  squared  effects  of  the  ignored  factors  are 

A 

absorbed  into  the  variance  of  3j •  Hence,  even  when  the  responses  are 

A 

observed  without  experimental  error,  V(3^)  may  unduly  large. 

Because  each  factor  is  at  two  levels,  the  standard  F-test  to  test 

Hq:  3  *  0  vs  H  :  ^  0  is  equivalent  to  a  simple  two-sampla  t-test 

th 

between  the  high  (+1)  and  the  low  (-1)  levels  of  the  j  factor.  The 
corresponding  test  statistic,  t ^ ,  is  given  by 

t  =  B  /  [SSE  /H(N  -  2))**  (3.11) 

where  SSE^  is  the  familiar  analysis  of  variance  notation  for  the  error 
sum  of  squares  of  factor  j.  Computationally, 

SSE  -  E (y  -  y.,)2  +  E(y  -  y  )2  (3.12) 

J  H  1  J  L  J 

where  the  first  (second)  summation  is  taken  over  the  N/2  observations 
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at  the  high  (low)  level  of  the  j  factor.  Other  useful  identities 
include 


N 

SSE .  =  Zy.2-  Ng.2 

3  i-1  3 


Ny2  , 


SSE 


and 


i 

SSE 


N 


E  3  B  -1‘ 
i-1  1  1  J 


J  v’v-mSj2 


(3.13) 

(3.14) 

(3.15) 


where  y  is  the  overall  mean  of  the  responses  and  v  ■  E  g.x,  • 

i=l 

Since  is  a  two-sided  alternative,  it  is  reasonable  to  reject 
if  the  observed  value  of  ]t^|  equals  or  exceeds  the  upper  100(1  -  a/2) 
percentage  point  of  a  t  distribution  having  (N-2)  degrees  of  freedom. 
Denote  this  critical  value  by  t(N-2;  a/2).  However,  it  should  be 
emphasized  that  although  t^  has  the  general  form  of  a  t  statistic,  the 
distribution  of  tj  under  (i.e.,  when  factor  j  is  inactive)  is  not  a 

0  vs  H^s  gj  ^  0 
outlined  here  is  not  an  exact  testing  procedure  in  that  the  true 
significance  level  of  the  test  may  differ  from  a. 

In  the  following  section,  results  based  on  the  empirical  distribution 
of  tj ,  obtained  by  Monte  Carlo,  are  compared  with  those  based  on  an 
approximating  t  distribution.  As  will  be  seen,  t^  is,  in  general,  fairly 
well-approximated  by  a  central  t  variable  under  Hq  and  by  a  noncentral  t 
variable  under  H^. 


t  distribution.  Consequently,  the  test  of  H^:  g^ 
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IV.  PERFORMANCE  EVALUATION  OF  THE  RB/PB  STRATEGY 


Define 


1,  if  Hq:  gj  =  0  is  rejected 
0,  if  3^  *  0  is  not  rejected, 

1,  if  factor  j  is  active 
0,  if  factor  j  is  inactive. 


Observe  that 


E(A)  =  Id,  E(V . )  , 
j-1  J  J 

K 

e(i)  -  z  a  -  dd  Edp 

j=l  J  J 


(4.1) 


(4.2) 


Accordingly,  to  evaluate  E(A)  and  E(I),  and  subsequently  E(F)  and  E(Q), 
it  suffices  to  calculate  E(V^)  for  all  j.  Note,  however,  that 

EOy  =»  P(Vj  -  1)  -  P  {|tj|  >  t(N-2;  a/2)}. 

If  the  exact  sampling  distribution  of  t^  was  known,  then  this  probability 
could  be  evaluated  directly  and  precisely.  Unfortunately,  the  distribution 
of  tj  is  intractable.  Performance  evaluation  must  rely,  therefore,  on  an 
approximating  distribution  or  on  Monte  Carlo  estimation. 

Motivated  by  large-sample  properties  of  t  ^ ,  it  is  possible  to  obtain 
a  fixed  sample  size  approximation  to  the  distribution  of  tj .  This 
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approximation  states  that  for  fixed  N,  tj  has  an  approximate  noncentral 
t  distribution  with  N  -  2  degrees  of  freedom  and  noncentrality  parameter 

&i  -  ife  /  [W  -  Sj2!5*.  (4.3) 

Hence,  if  denotes  a  random  variable  having  such  a  distribution, 

then 

K(V  )  s  p{|TN_2(6j)l  1  t(N"2?  a/2)}-  (4<4) 

Denote  the  probability  in  (4.4)  by  <p(6j).  Of  course,  when  factor 
j  is  inactive,  the  corresponding  probability  equals  a,  since  T«2(0) 
has  a  central  t  distribution.  Furthermore,  when  all  active  factors 


active  factor  because  of  symmetry.  Thus 

E(A)  *  kif/(6)  (4.5) 

and  E(I)  ss  (K  -  k)a  (4.6) 

where  6  *  (N/(k  -  1)]  2. 

The  quantity  ^ (6 )  is  simply  the  power  of  the  test  for  detecting 
an  active  factor.  Substituting  (4.5)  and  (4.6)  into  (3.1),  (3.2)  and 
(3.3)  yields 

E(F)  »  6)  (4.7) 

and  E(Q)  »  [cK  +  kip (6)  +  (K  -  k)a  +  2.5} /B(K).  (4.8) 

In  order  to  examine  the  quality  of  the  approximations  provided  by 
the  noncentral  t  distribution,  a  Monte  Carlo  computer  program  was  used 
to  obtain  empirical  estimates  for  comparison  in  a  number  of  cases. 
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On  the  whole,  the  approximations  performed  quite  well,  even  for 
relatively  small  values  of  N,  As  an  illustration,  Tables  7  and  13 
in  the  Appendix  contain  the  Monte  Carlo  estimates  and  the  approxi¬ 
mations  provided  by  the  noncentral  t  distribution,  respectively, 
when  K  «  120  and  k  »  10.  The  entries  in  these  tables  correspond  to 
the  126  RP(c,a)  strategies  defined  by  c  =  .1,  .2,  ...,  .8,  .9  and 

a  =  .05,  .10, . 65,  .70  .  As  can  be  seen,  the  results  agree 

fairly  closely  even  when  c  =  .1  . 

In  all,  performance  data  for  twelve  specific  cases  of  (K,k)  is 
tabled  in  the  Appendix.  For  each  of  three  K  values  (60,  120,  and 
240)  examined,  four  values  of  k  (k  =  p*K,  where  p*  =  2/60,  3/60, 

5/60,  and  8/60)  were  studied.  However,  because  the  Monte  Carlo 
procedures  became  prohibitively  expensive  for  large  N  and  K,  it 
was  not  feasible  to  simulate  over  the  full  range  of  c  for  all  cases. 

Thus,  the  performance  tables  in  the  Appendix  (except  for  K  =  120  and 
k  =  10)  consist  of  both  Monte  Carlo  estimates  (for  smaller  c)  and 
approximated  values  (for  larger  c)  of  E(A),  E(R),  E(F),  and  E(Q). 

Each  table  caption  indicates  the  source  of  the  entries. 

In  general,  E(F)  is  larger  and  E(Q)  is  smaller  for  a  smaller  p*. 

Thus,  as  intuition  would  suggest,  there  is  more  to  be  gained  from  the 
RB/PB  screening  strategy  when  the  proportion  of  active  factors  is 
smaller.  Furthermore,  for  fixed  p*,  both  E(F)  and  E(Q)  increase  as 
either  a  or  c  increases.  An  increase  in  the  significance  level,  a, 
results  in  a  less  stringent  rejection  rule  so  that  on  the  average  more 
factors,  both  inactive  and  active,  reach  stage  two,  hence  increasing 
E(F)  and  E(Q).  An  increase  in  the  number  of  first  stage  runs,  c,  increases 
directly  both  E(Q)  and  the  power  of  the  test,  hence  E(F). 
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For  fixed  c,  E(F)  Increases  from  0  to  1  and  E(Q)  Increases  from  cK/B(K) 
to  1  +  (cK)/B(K)  as  a  Increases  over  the  Interval  [0,1],  Therefore,  no 
matter  what  value  of  c  Is  decided  upon,  by  adjusting  a,  K(P)  can  be  Increased 
or  E(Q)  can  be  decreased  at  the  expense,  of  course,  of  Increasing  E(Q) 
or  decreasing  E(F),  respectively.  Consequently,  In  practice,  selecting  the 
c  and  a  to  actually  employ  will  depend  on  performance  trade-offs  the 
analyst  is  willing  to  make. 

To  illustrate  how  the  selection  of  a  RP(c,ot)  strategy  might  occur, 
suppose  K  »  240  factors  have  been  suggested  for  analysis,  but  only  about 
12  are  expected  to  have  an  appreciable  effect  on  the  response.  Using 
the  data  of  Table  10  (K  ■  240  and  k  ■  12),  contour  plots  of  E(F)  and 
E(Q)  as  functions  of  c  and  a  were  constructed  to  graphically  Illustrate 
the  available  trade-offs.  These  plots  were  overlaid  and  are  exhibited 
in  Figure  2.  Because  of  budget  considerations,  suppose  further  that 
it  is  desired  to  employ  the  RP(c,a)  strategy  that  maximizes  E(F)  subject 
to  E(Q)  £  .5.  For  all  practical  purposes,  as  is  apparent  from  Figure  2, 
any  RP(C,a)  strategy  corresponding  to  the  segment  AB  should  suffice.  For 
these  strategies  E(F)  is  roughly  .85. 

As  a  final  note,  it  is  instructive  to  rewrite  6  and  (4.8)  as 

6  *  (c/p*)*4  (4.9) 

and  E(Q)  -  c  +  p*ij>(6)  +  a(l  -  p*)  +  (2.5/K)  (4.10) 

where  (4.10)  follows  from  (4.8)  by  dividing  by  K  instead  of  B(K).  Recall 
that  B(K)  and  K  can  differ  by  at  most  four. 

Equations  (4.9)  and  (4.10)  together  with  (4.6)  show  basically  how 
c,  a,  and  p*  will  affect  performance.  For  instance,  increasing  6,  which 
is  equivalent  to  increasing  c  or  decreasing  p*,  increases  *K6),  i.e.,  E(F). 
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Overlay  of  Contour  Ploti 
E(Q),  Denoted  by  Dashed 


Equation  (A. 8)  also  illustrates  that  E(F)  depends  on  K  and  k  essentially 
only  through  p*.  This  is  further  evidenced  by  the  data  in  the  Appendix. 
Thus,  the  contour  plot  of  E(F)  in  Figure  2  could  be  used  as  a  reasonable 
approximation  for  other  K.  and  k  cases  in  which  k/K  ■  .05.  Finally,  as 
already  noted,  increasing  c  or  a  will  Increase  E(Q),  although  the  effect 
of  increasing  p*  is  not  directly  discernible*  Data  in  the  Appendix 
Indicates,  however,  that  E(Q)  increases  with  p*. 


V.  CONCLUDING  DISCUSSION 


This  report  evaluates,  in  cases  of  no  random  error  (or  equivalently, 
where  the  effects  of  active  factors  are  known  to  be  large  enough  to  ignore 
random  error),  the  factor  classification  and  testing  cost  characteristics 
of  the  RB/PB  screening  strategy.  Because  these  are  conflicting  objectives, 
an  overall  performance  criterion  for  comparing  different  RB/PB  strategies 
must  consider  these  characteristics  jointly.  Ultimately,  in  a  given 
situation  it  is  the  simulation  user  who  must  decide  on  the  trade-offs 
to  be  made.  Accordingly,  the  results  of  this  report  should  serve  as  a 
practical  guide  in  decisions  about  the  possible  use  and  choice  of  an 
RB/PB  strategy. 

In  section  IV,  comparison  with  Monte  Carlo  estimates  showed  that 
the  noncentral  t  distribution  provided  an  adequate  approximation  to 


E(Vj)  when  all  active  factors  have  the  same  absolute  effect,  A.  Although 
no  empirical  estimates  were  calculated  in  the  case  of  an  arbitrary  3., 


it  seems  reasonable  that  will  still  provide  an  adequate  approximation 

to  E(Vj).  However,  active  factors  should  probably  not  be  treated 
symmetrically  since  their  magnitudes  may  differ.  In  such  cases,  E(F) 


could  be  defined  as 


E(F)  «  Ed,  f 3. |  E(V  )/  E  |B 
j-1  J  2  j  j=l  j 


(5.1) 


where  d^  is  defined  in  section  IV.  Note  that  0  E(F)  <_  1  and  reduces 
to  (3.1)  when  all  active  factors  have  the  same  absolute  effect. 
Performance,  E(F)  and  E(Q),  of  the  RP(c,a)  strategy,  therefore,  can  be 
easily  extended  to  an  arbitrary  3,  by  letting  E(Vj)  ~  i£(<$j)  and 
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substituting  this  approximation  into  (4,1),  (4*2),  (3.2),  (3,3),  and  (5,1), 
As  a  by-product,  the  results  of  this  report  help  to  shed  some  light 
on  the  controversy  surrounding  random  balance  experimentation.  For 
instance,  let  RB(c,a)  denote  the  screening  strategy  based  solely  on  a 
random  balance  design  having  N  ■  cK  runs.  Suppose  again  that  each  factor 
is  tested  for  Importance  separately  with  a  standard  F-test  at  the  a  level 
of  significance.  In  the  case  of  no  random  error  where  all  active  factors 
have  the  same  absolute  effect, 

E{f|rB(c,ci)}  *  E{F|RP(c,a)}  *iM<5) 

E{q|RB(c,cx)}  »  cK/B (K) , 
and  Efj ]RB (c,a) }/(K  -  k)  »  1  -  a 

where  J  denotes  the  number  of  inactive  factors  identified  correctly. 

Note  that  the  classification  efficiency  of  the  RB(c,Ct)  strategy  for 
detecting  the  active  factors  is  the  same  as  that  of  the  RP(c,oc)  strategy, 
but  that  the  relative  testing  cost  of  the  RB(c,a)  strategy  is  smaller 
since  there  is  no  follow-up  stage.  Of  course,  deleting  the  PB  second 
stage  introduces  the  possibility  of  classifying  as  active  an  inactive 
factor.  In  this  case,  overall  performance  will  require  evaluation 
against  all  three  of  the  screening  objectives  listed  in  section  III. 
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APPENDIX 


! 

I 

I 


The  following  pages  contain  the  results  of  the  calculations  for  the 
twelve  specific  cases  of  (K,k)  examined.  Corresponding  to  each  value  of 

c  (c  -  .1,  .2, . 9)  and  a  (a  -  .05,  .10, . 70)  is  a  block  of  four 

numbers.  The  numbers  within  a  block  are  arranged  as  follows: 

E(A),  the  expected  number  of  active  factors  correctly 
identified, 

E(R) ,  the  expected  total  number  of  runs, 

100  E(F),  where  E(F)  is  the  classification  efficiency 
for  active  factor  detection, 

and  100  E(Q),  where  E(Q)  is  the  relative  testing  cost. 
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Table  5:  Performance  Measures  for  K  >  120  and  k  -  4.  (Entries  corresponding 
to  values  of  c  greater  than  4  are  asymptotic  values.) 
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Table  6:  Performance  Measures  for  K  -  120  and  k  -  6.  (Entries  corresponding 
to  values  of  c  greater  than  4  are  asymptotic  values.) 
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Table  8:  Performance  Measures  for  K  -  120  and  k  -  16.  (Entries  corresponding 
to  values  of  c  greater  than  4  are  asymptotic  values.) 
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Table  9:  Performance  Measures  for  K  ■  240  and  k  *»  8.  (Entries  corresponding 
to  values  of  c  greater  than  3  are  asaptotlc  values.) 
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Table  10:  Performance  Measures  for  K  -  210  and  k  -  12.  (Entries  corresponding 
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